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Reviewing the research on teachers’ diagnostic competences shows that most findings 
focus on the correspondence between teachers’ diagnostic judgments and students’ 
actual achievement, while cognitive processes and cognitive resources of teachers in 
diagnostic situations have been examined much less. We intend to extend this state of 
research from a domain-specific point of view by empirically identifying and 
theoretically describing processes and resources of mathematics teachers while 
judging tasks and students’ solutions. In an interview study with expert teachers and 
mathematics educators (n=6) it was possible to deduce typical steps in a diagnostic 
process to identify resources (i.e. aspects of teacher knowledge) they relied on. 


INTRODUCTION 


In mathematics teaching we find many different diagnostic situations, which can be 
characterized according to their position in the learning process and their respective 
objectives (e.g. Ingenkamp & Lissmann 2008; Wiliam 2007): 


e §=_[nitial assessment aims at gaining information about the students’ conditions 
for future learning (e.g. previous knowledge of students). 

e Formative assessment is needed for supporting individuals or for adapting 
instructional choices during the learning process. 

e Summative assessment is needed for assessing learning results and can be 
used for certification or placement of students. 


(Depending on the authors the terms ‘assessment’ and ‘diagnosis’ are considered either 
synonymous or contrasting in certain aspects. In this paper we assume no difference). 
Diagnostic situations can also be differentiated by the level of formality: In addition to 
formal diagnostic tests, there are also informal diagnostic situations which influence 
instruction. In mathematics teaching such diagnostic situations are often linked to the 
activity of working with tasks, e.g. (1) Teachers analyse and select tasks with respect to 
their potential diagnostic value and (11) teachers evaluate students’ solutions to a task. 


Current and recent research focuses on the precision of teachers’ diagnostic 
judgements (dubbed the ‘veridicality-paradigm’) (cf. Hoge & Colardaci 1989, 
Sudkamp, Kaiser & Moller 2012), while many questions regarding the cognitive 
processes of teachers during the assessment process and the domain specificity of 
diagnostic competence remain unsettled (Schrader, 2011). In a similar way that Ball, 
Thames & Phelps (2008) investigated mathematical knowledge for teaching by 
analysing teachers activities, we intend to create some insights into teachers’ 
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diagnostic competencies by analysing their cognitive processes and their use of 
resources during assessment. 


THEORETICAL FRAMEWORK 
Concepts of diagnostic competence 


Diagnostic competence most often is defined as the ability of a person to judge people 
appropriately (Schrader, 2011) and measured by numerical indicators for the precision 
of such diagnostic judgements. Three aspects of precision are frequently studied, each 
of them being related to specific diagnostic activities and situations (Spinath 2005, 
Lorenz & Artelt 2009, Schrader & Helmke 1987): (1) The judgement of a Jevel of an 
attribute of a student or a task relates to the situation of selecting tasks with an 
appropriate content or level of difficulty. One can ask if teachers underrate or overrate 
such attributes. (2) Judging the variance of some attributes within a group of students 
is necessary for deciding about individualisation strategies. Finally (3) correctly 
estimating the rank (a) of the difficulty of tasks or (b) the abilities of students can tell 
something on the use of content knowledge for selecting tasks or the knowledge on the 
relative strengths and weaknesses of the class. It seems obvious that the numerical 
precision of such judgments can only be regarded as an indicator for diagnostic 
competence at work. Within this approach knowledge about the structure and the 
influencing factors of diagnostic competence (pertaining to the task, the student, the 
context or the teacher) is based on studying the reasons for judgment biases (Sudkamp, 
Kaiser & Moller 2012). 


Still there are many open questions left, such as in which way teachers generate 
diagnostic judgements in the pedagogical context. There is a lack of understanding of 
cognitive processes of teachers guiding their judgement. Also the domain-specificity 
or even topic-specificity of diagnostic competence and how diagnostic competence is 
composed would be of interest. By correlating the above-mentioned indicators Spinath 
(2005) showed that diagnostic competence should not be considered as general ability 
but rather as construct that consists of several sub-competences. Still we do not possess 
any fairly coherent theoretical model of diagnostic competence and empirical evidence 
for it (Schrader 2011; Anders et al. 2010). 


For mathematics education it is a fruitful task to contribute to a better understanding of 
the processes and the knowledge connected to diagnostic situations with respect to the 
domain of mathematics. This can be seen as embedded within the broader challenge of 
constructing a theory of teacher knowledge in mathematics. For example, within the 
framework of Ball et al. (2008) competences needed for diagnostic activities can be 
located in several areas: Common content knowledge (CCK) is needed to evaluate the 
correctness of a student’s solution for instance, specialized content knowledge (SCK) 
is used for example to vary the degree of difficulty of tasks and knowledge of content 
and students (KCS) helps to understand students (mis-)conceptions and approaches. 
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Approaches for modelling diagnostic processes 


Understanding cognitive processes in diagnostic situations can also be seen as a 
question within the field of research on expertise. Here one can find several models for 
cognitive processes in diagnostic situations which can be used as a framework for 
further research: Croskerry (2009) proposes a model for diagnostic reasoning in the 
medical context by integrating previous efforts of promoting diagnostic competence of 
physicians: (1) the intuitive approach leaning on experience and gestalt effects and (2) 
the analytic approach using knowledge and systematic information gathering. To 
describe diagnostic judgements she proposes a dual process model (in the sense of 
Kahneman, 2003) where patterns are processed by an unconscious system and by 
rational processes of a conscious system which interact in specific ways (practice, 
override and calibration processes) to reach a diagnostic judgement. The fact that 
Croskerry (2009) calls this a “universal” model already indicates that this can be 
considered a broad framework which leaves many space for specification (such as by 
modelling the conscious system by critical thinking, training, logical competence etc.). 
Nickerson (1999) proposes a model to describe the process of rating other people’s 
knowledge. First a model of own knowledge is used as an anchor to describe the 
knowledge of others (default model). In several steps this model is refined by including 
information on the particularity of one’s own position, on the random other and on 
more and more information on specific others. This way the process of gaining insight 
in other people’s knowledge can be seen as an alternation of anchoring and adjustment 
(Tversky & Kahnemann 1974). In this model Nickerson can explain frequent 
tendencies of overestimating knowledge of others. Nickersons model appears to be 
very general and especially refers to factual knowledge. It should be transferred into 
pedagogical context with caution. Morris et al. (2009) on the other hand construct a 
model very specific to a diagnostic situation in mathematics teaching. They show that 
“unpacking” the sub-goals of a task can be considered an important facet of diagnostic 
competence with regard to the planning and evaluation of learning processes. The 
ability to decompose mathematical content within a task can be useful in diagnostic 
situations to locate students’ mistakes. However is doubtful if the “unpacking 
competence” is enough to master diagnostic situations which require identifying 
deficient conceptions of students, since misconceptions (such as the “division makes 
smaller’ error in calculating with fractions) cannot be deduced by analysing correct 
solution processes. 


These examples of very different scope show, that there are indeed different 
frameworks available for modelling cognitive processes and knowledge resources of 
teachers during diagnostic activities. To substantiate these models it seems desirable to 
have a concrete picture of cognitive processes of mathematics teachers. It is our goal 
not to test these general models but to create knowledge on processes in the concrete 
domain of mathematics teaching that can connect to the more general models and 
inspire further research in this area. 
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RESEARCH QUESTIONS 


In our study we focus on informal diagnostic situations and teachers working with 
mathematical tasks in a diagnostic way, such as when judging tasks or evaluating 
students’ solutions. These diagnostic situations often occur when tasks have to be 
selected and embedded in existing material or when the teacher has to react towards 
students mistakes spontaneously. For an in-depth investigation of teachers’ diagnostic 
competence in these situations we assume a double focus on processes and on 
resources during the formation of diagnostic judgments and pursue the following 
research questions: (1) What kind of processes can be identified in teachers’ diagnostic 
judgements? (2) What kind of knowledge do teachers rely on during these processes? 
By these questions we intend to create a deeper understanding of diagnostic processes 
but also to further clarify possible components of diagnostic competence of 
mathematics educators. A long-term objective connected with our research is to derive 
consequences for teacher education and professional development. 


DESIGN OF THE STUDY 


As a method to gain information on cognitive processes and knowledge of teachers we 
decided to capture their reasoning by means of two phased think-aloud interviews 
(Ericsson & Simon 1993). In the first phase we initiated diagnostic processes by first 
presenting two tasks and afterwards three students’ solutions to each task and asking 
the participants to evaluate each of them. In the second phase teachers had to reflect 
their own diagnostic process by describing the process and additionally by giving 
reasons for their judgement. By this combination of parallel and retrospective think we 
expected to capture a large part of the relevant processes. 


As participants we chose three experienced mathematics teachers and three scholars in 
mathematics education. The latter had experiences as mathematics teachers and as 
teacher educators (for at least three years in each of their professional phases) and so 
we could draw on practical experience and reflected theoretical knowledge likewise. 
The aim of selecting this sample was to find a maximum variety of different processes. 
Think-aloud-protocols of the diagnostic processes and the reflections of their own 
processes supplied the data for the analysis in the present study which amounted to 12 
evaluations of tasks and 36 evaluations of students’ solutions. For the interviews we 
chose the tasks from the topic “fractions”, because of the broad systematic knowledge 
about students’ conceptions, errors and misconceptions in this field. The tasks and the 
interview guideline were developed and optimized in a pilot study. The students’ 
solutions were selected so that they represented typical mistakes and frequent 
misconceptions. Figure 1 shows the tasks and solutions we used. 


In the first phase the participants had to analyse tasks. They were asked: “Which 
challenges do you see? Which difficulties do you expect?” Then the participants were 
given the three students’ solutions to each task and had to evaluate them by answering 
the question: “Which conclusions do you draw?” 
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Figure 1: Tasks and students’ solutions (from: Wartha, 2007). 


Interpretative content analysis techniques were used to analyse the data (Ericsson & 
Simon 1993, Mayring 1983) with the objectives were to reconstruct types of diagnostic 
processes and to generate a theoretical overarching structure. In the analysis we first 
focused on assessment processes (see research question 1). In the next step we 
analysed the same data with a focus on the kinds of knowledge underlying these 
processes (see research question 2). 


RESULTS 


We present some exemplary results of the two interpretative cycles described above. 
Focussing on diagnostic processes (see research question (1)) resulted in more than 15 
Processes, of which we present three important ones. Table 1 shows the name of the 
process (code), a description of the code and excerpt of an interview to illustrate the 
category. 


Code Description Representative teacher statement 
Standard Design a solution ” , ook a 
eAinnGa eens [...] you can solve it by division. 

Discover and name “1/4 is bigger than 1/3. This is typical. 
Identify deficits an incorrect When the numbers are in the denominator. 
approach. With bigger and smaller.” 
Identify Discover and name “T...] this is great. He writes down the 
strengths skills. number 2400 as fraction.” 


Table 1: Excerpt of identified assessment processes. 


The category “standard solution” refers to the process of designing a solution on your 
own or mentioning a common solution approach by its name. “Identify deficits” refers 
to recognizing an incorrect approach in a student solution. Finally to “identify 
strengths” means to see students’ competences in their solutions. When analysing the 
same data with a focus on the kinds of knowledge underlying these processes (see 
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research question (2)) we found among others the following knowledge categories: In 
the first example in Table 1 (see above) the interviewee refers to the mathematical 
correctness and therefore draws on his mathematical knowledge. In the second 
example the interviewee explicitly refers to a typical mistake and therefore uses a 
component of pedagogical content knowledge that refers to systematic knowledge 
(gained in educational research). In the third example one can see a reference to a 
concept of a fraction as a rational number — although it remains unclear whether this 
should be assigned to explicit knowledge on students’ development of number 
concepts or merely to the recognition of a mathematical fact. 


When analysing the results of the coding process (which could only be indicated by 
few examples here) it is possible to develop a “bigger picture”: Some of the processes 
are essential; they show up frequently and can be interpreted as steps in an assessment 
process. In every step different qualities of individual processes were observable. 


Standard 
solution 


Give possible Understand 
reasons for students 
errors solution 


Identify Identify 
deficits strengths 


Figure 2: Idealized five-step model for the diagnostic process. 


Figure 2 shows an idealized model of steps during informal assessment: The initial 
point often is a standard solution or an approach. Then own solutions are compared 
with students’ solutions. Thereby strengths and deficits of the solution can be 
identified. The last step is to find a (hypothetical) reason (or several reasons) for errors 
— if they occur. As a very common strategy across all steps we observed that 
interviewees spontaneously decompose tasks or solutions and to analyse them step by 
step — just as Morris et al (2009) advise the participants in their study. 


The cognitive resources the interviewees rely on when moving through the diagnostic 
process as described above can be characterized as different types of knowledge: We 
could identify content knowledge (CK) and pedagogical content knowledge (PCK). 
For example, the participants of our study used knowledge on mental models of 
mathematical concepts, on typical errors and on typical misconceptions, Furthermore 
mathematical correctness was evaluated and student strategies were identified. 


CONCLUSIONS AND DISCUSSION 


The main objective of our study was a deeper understanding of diagnostic processes. 
Although such processes followed quite individual patterns, they could be categorized 
as different types of “steps in the diagnostic process”. Furthermore it became evident 
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that the participants showed different degrees of flexibility, for example in the number 
of possible approaches to solve a task they mentioned (often combined with more than 
one representation). In future analyses this flexibility may serve as an indicator for the 
quality of the diagnostic process and/or the diagnostic competence of the teacher. To 
clarify this connection remains an open question for further study. 


Another objective was to identify types of knowledge, which teachers use while 
forming their diagnostic judgement — this amounts to delineate different components 
of diagnostic competence. A provisional interpretation of our results with regard to 
aspects of diagnostic competence is that three different aspects can be identified: 
(1) Knowledge: the use of content knowledge (CK) as well as pedagogical content 
knowledge (PCK) was observable. (2) Abilities: we observed the ability to decompose 
mathematical tasks but also to analyse tasks and solutions step by step and the ability to 
take the students’ perspective. (3) Attitudes: we state that a kind of readiness for 
assessment is necessary e.g. for taking students perspective — although this aspect did 
not emerge directly by our systematic analysis but is inferred rather generally from our 
experience during the interviews. 


We regard our results as modest extensions to theoretical frameworks which only 
partially focus on diagnostic competence, First our results regarding aspects of 
knowledge can be integrated into the theoretical framework of Ball et al. (2008) but 
still need further foundation, e.g. by efforts to quantitatively measure the aspects 
described here. Second, the decomposing of tasks and students solutions found within 
our study can be considered close to the research by Morris et al. (2009). However, 
while Morris et al. refer only to teachers “unpacking” mathematical concepts one 
should also consider the process of teachers identifying misconceptions that cannot be 


deduced by starting from correct mathematical concepts. 


Finally our research also uncovered certain differences in diagnostic processes of 
teachers with different levels of experience. For example, in our analyses some striking 
differences showed up, which also should be investigated further: Experts (with a 
scholarly background) seem to use a variety of different approaches to analyse a task. 
They appear to be more focused on strengths in their assessment than teachers. Experts 
draw more explicitly on subject-based knowledge (PCK). Because of the sample size 
of our study these differences can be seen as tendencies only. They should be regarded 
as hypotheses which need a more rigorous treatment and may be tested in a different 
design. 
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